Methods and apparatus for automated true object-based image analysis and retrieval

ABSTRACT

An automated and extensible system is provided for the analysis and retrieval of images based on region-of-interest (ROI) analysis of one or more true objects depicted by an image. The system uses an ROI database that is a relational or analytical database containing searchable vectors representing images stored in a repository. Entries in the ROI database are created by an image locator and ROI classifier that locate images within the repository and extract relevant information to be stored in the ROI database. The ROI classifier analyzes objects in an image to arrive at actual features of the true object. Graphical searches may also be performed.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 12/584,894, filed Sep. 14, 2009, which is a continuation ofU.S. patent application Ser. No. 11/122,969, filed May 5, 2005, now U.S.Pat. No. 7,590,310, which claims the benefit of the filing date of U.S.Provisional Patent Application No. 60/568,412, filed May 5, 2004, theentire disclosures of which are hereby incorporated herein by reference.

FIELD OF THE INVENTION

This invention generally relates to query processing or searchtechniques for databases or files containing visual information orimages. More specifically, the present invention is directed to anautomated and extensible system for the analysis and retrieval of imagesbased on region-of-interest (ROI) analysis of one or more true objectsdepicted by an image.

BACKGROUND OF THE INVENTION

The proliferation of digital images and video has been a direct resultof more efficient methods for gathering digital imagery. From recordingdevices such as digital cameras and digital video recorders, todigitizers for CCTV and broadcast media, to 2-D home and office scannersfor producing digitized versions of photographs, the myriad digitalrecording methods has created an explosion of digital media.

For purposes of the present invention, the term “image” will signify atwo-dimensional digital representation of a scene, typically created anddigitized by an image capture device, signifying a standalone event or amomentary representation of a sequence of events. The term “object” willrefer to a uniquely identifiable physical thing or item at leastpartially depicted in an image. The term “content” will refer to thesubject matter of an image.

Along with the countless forms of digital imagery, many differentmethods have appeared that attempt to organize and structure the digitalrepositories in such a way that permit retrieval of images in astructured or semi-structured fashion. Unfortunately, providing somestructure to the imagery requires conformance to a certain format,necessitates an input process that structures the information, orrequires human interaction to annotate the imagery for later retrieval.The most popular examples of image search engines, such as Alta Vistaand Google, for example, rely exclusively on annotations and/or filenames associated with the images as the only basis for determiningwhether that image contains desired visual information.

Some image search systems have attempted to overcome the “inputrequirements” of an annotated image repository by automating theextraction of image features for objects depicted within the data set ofthe image that results in significantly-reduced representations of thecontents of the image repository, thereby decreasing the search time fordesired images. U.S. Pat. No. 6,438,130, for example, describes amulti-level filtering technique for reducing the data for content-basedimage searches. U.S. Pat. No. 6,271,840 discusses a browser interfacefor displaying reduced versions of images created by an integrated pagerenderer.

Several techniques exist for the indexing, categorization, anddescription of images that will be retrieved via a multitude ofsearching algorithms. Typical image indexing techniques utilize basicattributes like color, shape and texture to describe images or regionsof interest within those images.

U.S. Pat. No. 6,778,697 discloses a color image processing method forretrieving a color feature descriptor for describing color features ofan image. The color image processing method includes obtaining colorvectors of an input image, classifying the color vectors to obtaindominant colors of the input image and their ratios, and representingthe dominant colors and their ratios as a color feature descriptor ofthe input image to allow fast search and retrieval. U.S. Pat. No.6,411,953 provides a perceptually-based system for pattern retrieval andmatching which uses a predetermined vocabulary comprising one or moredimensions to extract color and texture information from an imageselected by a user. The system then generates a distance measurecharacterizing the relationship of the selected image to another imagestored in a database, by applying a grammar, comprising a set ofpredetermined rules, to the color and texture information extracted fromthe selected image and corresponding color and texture informationassociated with the stored image.

U.S. Pat. No. 5,802,361 discloses an analysis and retrieval system thatuses image attributes to retrieve selected images from a database. Thepatent provides a fundamental description of the process required tosearch large image archives for desired images.

U.S. Pat. Nos. 5,893,095 and 5,911,139 describe a system forcontent-based search and retrieval that utilizes primitives to operateon visual objects. The fundamental approach taught in this patent is thecomparison of visual objects and the ability to identify similarities invisual perception of the images. Another approach that utilizessimilarity of images is described in U.S. Pat. Nos. 6,463,432 and6,226,636 ('636). The '636 patent discloses a system that builds adatabase which stores data corresponding to a plurality of images bydividing each image into several regions. Then, the system calculates ahistogram of each region. The database may then be used to determineimages which are similar to a query image. U.S. Pat. No. 6,502,105teaches a system that builds a database of image-related data byinputting a plurality of images, and for each image: dividing the imageinto a plurality of regions and generating a graph based on the regions,and storing data for the graph in the database. The database may then beused to determine whether a query image is similar to one or more of theplurality of images by dividing the query image into regions andgenerating a graph based on the regions, and comparing the generatedgraph to other graphs in the database that correspond to the pluralityof images. U.S. Pat. No. 6,240,424 teaches a method and apparatus forclassifying and querying a database of images, in which the images inthe database are classified using primary objects as a clusteringcenter.

U.S. Pat. No. 5,983,237 discloses an image analysis system consisting ofa visual dictionary, a visual query processing method, an imagedatabase, a visual query processing system, and the creation of featurevectors. This system discusses the processing of imagery as an integralpart of storing the image in the database, whereby both annotated textand feature vectors can be associated with the image. The subsequentsearching of images can occur via text searches based on the annotationand on visual searches based on the feature vectors of that image.

U.S. Pat. No. 6,084,595 discloses a method of utilizing indexedretrieval to improve computational efficiency for searching largedatabases of objects such as images. The premise of the approach is tonegate the requirement of a query engine to search all vectors for animage within a feature vector database.

U.S. Pat. No. 6,389,417 discloses a more advanced approach to imageretrieval based on determining characteristics of images within regions,not for an entire image. The system is limited, however, to using queryregions based on an image supplied by the user, thus limiting theinvention to finding images that contain regions similar to a targetregion within a provided image.

U.S. Pat. No. 6,566,710 reveals a graphical search technique based onjoint histograms. The invention discloses usable methods for achievingmatches and/or near-matches of images based on similar joint histograms.

U.S. Pat. No. 6,574,378 describes a search and retrieval system based on“visual keywords derived using a learning technique from a plurality ofvisual tokens extracted across a predetermined number of visualelements.” As with the other systems and methods described above, thispatent utilizes the analysis of the image and its contents, with nomention of understanding the various elements or objects that comprisethe components within the scene.

U.S. Pat. No. 6,574,616 attempts to improve on image understanding ofthe preferences of a given user by using probability functions. Thesystem suffers from the requirement that images must be iterativelyselected by a user, with more images presented based on the iterativeselections and the probability functions of the images.

U.S. Pat. No. 6,584,221 describes advances in image search and retrievalby separating images into regions of interest and making searches basedon color and texture. While image color and texture of a region ofinterest are used in this patent, no attempt is made to determine thephysical attributes of the objects depicted in the images.

U.S. Pat. No. 6,611,628 reveals an image feature-coding scheme thatassigns a color and a relative area to regions within an image frame.This approach utilizes image and frame color for the feature coding,with no attempts made to interpret the actual color of the objectsdepicted in the images.

U.S. Pat. Nos. 5,579,471, 5,647,058, and 6,389,424 describe varioustechniques for creating high dimensional index structures that are usedfor content-based image retrieval. U.S. Pat. No. 5,647,058 discloses ahigh dimensional indexing method which takes a set of objects that canbe viewed as N-dimensional data vectors and builds an indexed set oftruncated transformed vectors which treats the objects likek-dimensional points. The set of truncated transformed vectors can beused to conduct a preliminary similarity search using the previouslycreated index to retrieve the qualifying records.

U.S. Pat. No. 6,563,959 describes a perceptual similarity imageretrieval system in which images are subdivided into spots with similarcharacteristics and each image is represented by a group of spotdescriptors.

Techniques have been described for creating a so-called blobworld orregional representation of an image as part of a region-based imagequerying. See, Carson et al., “Region Base Image Querying,” Proc. ofIEEE CVPR Workshop on Content—Based Access of Images and VideoLibraries, 1997, Lui et al., “Scalable Object-Based Image Retrieval,”.pdf paper, Ozer et al., “A Graph Based Object Description forInformation Retrieval in Digital Image and Video Libraries,” .pdf paper,Fan et al., “Automatic Model-Based Semantic Object ExtractionAlgorithm,” IEEE Trans. on Circuits and Systems for Video Technology,Vol. 11, No. 10, October 2001, pp. 1073, and Ardizzoni, et al.,“Windsurf: Region-based image retrieval using wavelets,” Proc. of the1st Int'l Workshop on Similarity Search, September 1999, pp. 167-173.These references describe different techniques for the creation,segmentation or hierarchical arrangement of representations of an imageusing a region-based, blobworld representation of that image.

Large image repositories will undoubtedly contain imagery from amultitude of input sources. Any image search and retrieval mechanismmust be specifically designed to deal with the wide variation ofattribute types that will result from the myriad input sources. For arepository as complex and diverse as the Internet, a flexible imagesearch and retrieval engine should be able to interpret information fromthe original image independent of the type of input device or digitizerthat was used to create the image.

Computer vision and image processing techniques alone are not adequatefor robust image search and retrieval. In addition, implementing imageretrieval on a large scale requires automated techniques for analyzing,tagging and managing the visual assets. What is needed for large-scaleimage search and retrieval that is an automated system that can operateon unstructured data. This automated system must utilize techniquesbeyond traditional image processing in order to increase the hit rate ofimage retrieval that demanding users will require. Furthermore, thesystem must be capable of dealing with a constantly changing targetimage set wherein the system has little or no control over the contentplaced within the image search space.

BRIEF SUMMARY OF THE INVENTION

The present invention is an automated and extensible system for theanalysis and retrieval of images based on region-of-interest (ROI)analysis of one or more true objects depicted by an image. The systemuses a Regions Of Interest (ROI) database that is a relational oranalytical database containing searchable vectors that represent theimages stored in a repository. Entries in the ROI database are createdby an image locator and ROI classifier that work in tandem to locateimages within the repository and extract the relevant information thatwill be stored in the ROI database. Unlike existing region-of-interestsearch systems, the ROI classifier analyzes objects in an image toarrive at the actual features of the true object, instead of merelydescribing the features of the image of that object. Graphical searchesare performed by the collaborative workings of an image retrievalmodule, an image search requestor and an ROI query module. The imagesearch requestor is an abstraction layer that translates user or agentsearch requests into the language understood by the ROI query.

The ROI analysis of the present invention focuses on the actual featuresof an object, which are referred to for purposes of this invention asthe “true object” or “physical object,” instead of the features asrepresented by the content of an image. For example, if an image of anobject, such as a car, is shot at sunset, a person understands that thecolor features of that image will be distorted from the actual color ofthe true object. A red car may appear as a purple car or an orange carin the image, but the true physical object is still a red car. The ROIanalysis preferably utilizes a combination of techniques to arrive atthe features of the true object. In the case of color, for example, afuzzy logic color filter technique is utilized to deduce the true coloras a true or “physical” vector of a given object represented by a regionof interest. For purposes of the present invention, the actual qualitiesor characteristics of an object derived by the ROI analysis will bereferred to as the “physical feature vectors” associated with that truephysical object.

Regions of interest representing an object in an image can be describedin a multitude of types and formats. Examples of region of interestdescriptions include, but are not limited to bounding boxes, polygons,and masks. Entries in the ROI database have an ROI type field thatcontains the type of the region of interest, and an ROI descriptor fieldthat contains the description of the region of interest that is of theformat indicated by the type. Since the number of bytes needed todescribe a region of interest is quite diverse, the ROI descriptor fieldcan be a small as a few bytes and can be as large as several kilobytes.Other fields in the ROI data structure for each entry in the ROIdatabase are the physical color vector, physical shape vector andphysical texture vector. Other physical vectors are also allowed withinthe data structure, including physical size vector, physical locationvector and physical content vector.

Images will typically contain multiple regions of interest. Theinvention described herein allows search and retrieval at the region ofinterest level, not at the image level. This allows specific searches tobe performed for desired objects without regard to other items that mayappear in the imagery. This approach does not preclude searches forimages that contain multiple objects. It merely confines the searchcriteria to the physical object level, leaving multi-object searches toa hierarchical layer above the object search layer.

The image locator preferably contains automated routines thatcontinually search the repository for the existence of new imagery.Software utilized for searching can be web crawlers, spiders, or anyother type of intelligent agent that can continuously search anunstructured network for the existence of new or updated files.

Any graphical search request will be initiated either by an intelligentagent or by a user. The user search request can originate from inside oroutside the network on which the search will take place.

The systems described in the prior art disclosures have gone to greatlengths to categorize, classify, locate and retrieve imagery based onimage characteristics. Image attributes like texture, shape and colorhave proven to be fairly reliable in retrieving images based onselections of combinations of these attributes. The main drawback,however, has been the focus on utilizing image characteristics forsearch criteria.

As an example, assume an image exists for a red car. As long as theangle of the camera with respect to the car allows the image to show astrong shape component for a car, a shape search for car will yieldreasonable results. Furthermore, if the image were taken on a sunny daythe color in the image will contain a strong red component. If, however,the image were taken with the car sitting under a tree, the shapecharacteristic would be impacted by the outline of the tree, and thebright red finish of the car would be duller, and perhaps not even red.If the image's color space were RGB, the red component could possibly bethe weakest of the three components.

In this example, a traditional search for “red car” would probably notyield a hit on this object because the shape of the object wascompromised and because the color of the object was not well representedin the image. In contrast, the present invention represents searchableregions of interest not by their image attributes, but by their truephysical object attributes.

Oftentimes the process of capturing imagery will add noise (which canimpact the object's texture and/or shape) or will dramatically changethe representative color of the imaged object. Imprecise orill-controlled image capture techniques should not spoil the quality ofthe search and retrieval system. The present invention overcomes theseproblems by providing for an image search and retrieval system based onthe actual characteristics of the physical object depicted by the image,not based on the characteristics contained within the image.

Determining physical vectors or attributes of true physical objects is acompute-intensive operation. These computations, however, are part ofthe image categorization phase, not the retrieval phase. The systemdescribed herein is designed to retrieve information very rapidly byutilizing simple searches for object attributes that have been createdas part of the previously completed image categorization phase.

Large repositories like corporate intranets and the Internet have amultitude of sources for new and updated information. The systemdescribed herein utilizes a crawler, spider or other intelligent agentto constantly update the image categorization process of searchinformation for all imagery. A limitation of the invention, at leastwith today's technology, is the image search space is confined to thoseimages that have been processed by the image categorization classifier.A repository the size of the Internet might not be fully searchableuntil the classifier had been given sufficient time to locate andanalyze most of the images in the available search space, which couldrequire months of operation on a large farm of computers before acritical mass of image analysis had been performed to allow reasonablyrepresentative searches. It will be noted that the inputs to theclassifier, however, can be directed to concentrate on certain pocketsof information within the image domain to achieve the earliestfunctionality for the search and retrieval system.

Both the classification and the retrieval functionality of thisinvention are designed to operate on single computers, on multiplecomputers, or on a distributed network of computers. The system diagramsshow a single Region Of Interest (ROI) database, but the system worksjust as well with multiple databases populated at several pointsthroughout the network. If multiple ROI databases are utilized, there isno strict requirement that any synchronization exists between thedifferent versions. It is possible, however, to have a completelyseparate or a fully integrated software program that searches the otherROI databases for new or updated ROI entries. These entries could thenbe replicated into the local ROI database.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the process of transforming an object's image color spaceinto a physical color space, thus yielding the Physical Color Vector.

FIG. 2 shows the process of transforming an object's image shape spaceinto a physical shape space, thus yielding the Physical Shape Vector.

FIG. 3 shows the process of transforming an object's image texture spaceinto a physical texture space, thus yielding the Physical TextureVector.

FIG. 4 is a block diagram of the image search and retrieval system.

FIG. 5 shows the Region Of Interest data structure

FIG. 6 is a depiction of the Physical Color Vector.

FIG. 7 is a depiction of the Physical Shape Vector.

FIG. 8 is a depiction of the Physical Texture Vector.

FIG. 9 shows a flowchart of the Image Locator functionality.

FIG. 10 shows a flowchart of the Region of Interest Classifierfunctionality.

FIG. 11 shows examples of various color image representations of thesame physical object.

FIG. 12 shows an intermediary processing step for obtaining a PhysicalColor Vector for the object represented by the images of FIG. 11.

DETAILED DESCRIPTION

Referring to FIG. 1, the Region Of Interest (ROI) image colorrepresentation 15 is shown as a three-dimensional volume positionedwithin the image color space 10. The ROI image color 15 can bedetermined by a multitude of techniques including, but not limited to,difference filters, Hough transforms, fuzzy logic color filters, and avariety of multi-filter techniques. The image color space 10 can berepresented in a variety of models including, but not limited to CIEYUV,XYZ, Lab, HSV and HSL.

Upon achieving the proper description of the image color space 15, thecolor filter transform 20 will be used to determine properrepresentation of the color 35 in the physical object color space 30.The physical object color space 30 is a 3-D volume that describes therange of colors for physical objects. Within the physical object colorspace 30, there is a multitude of overlapping volumes 40 that representa portion of the physical object color space 30. These overlappingvolumes 40 are each represented by fields within the physical colorvector 50.

The fields in the physical color vector 50 may be single-bit ormulti-bit fields. When multi-bit fields are used, the available quantumlevels can represent: 1) the probability that the physical object is thedesignated color 40, 2) the percentage of the designated color volume 40contained within the ROI physical color representation 35, 3) thedistance from the center of mass of the ROI physical colorrepresentation 35 to the center of mass of the designated color volume40, or some other measure that signifies the relative strength that therepresentative color 40 contributes to the physical color of theoriginal true object.

FIG. 2 shows the transformation of an image shape 65 to a physicalobject shape 85 with the use of a shape transform filter 70. First, theshape of the ROI from the image 65 is described within the coordinatesystem that is the image shape space 60. The image shape space 60 willusually be a Cartesian coordinate system, although certain image typescould utilize projection or polar coordinate systems. The physical shapevector 90 will contain fields that represent two-dimensionalrepresentations of physical objects. The shape vector lookup table (LUT)75 contains projections that correspond to the various fields within thephysical shape vector 90. The shape vector LUT 75 will likely havemultiple entries for each field in the physical shape vector.

The fields in the physical shape vector 90 may be single-bit ormulti-bit fields. When multi-bit fields are used, the available quantumlevels can represent the probability that the physical object is thedesignated shape, that it contains feature aspects of the designatedshape, or that some other measure that signifies the relative strengththat the representative shape contributes to the physical shape of theoriginal object.

The physical object shape space 80 may or may not be expressed in thesame coordinate system as the image shape space 60. The selection ofcoordinate system for the physical object shape space 80 will depend onthe image type, the scene descriptors, the context type (single image,series of still images, digital video, etc.), etc.

FIG. 3 shows the transformation of an image texture 100 to a physicalobject texture 120 with the use of a texture transform filter 110. Thephysical texture vector 130 will contain fields that represent texturesof physical objects. The texture vector lookup table (LUT) 115 containsmathematical descriptions that correspond to the various fields withinthe physical texture vector 130. The texture vector LUT 115 will likelyhave multiple entries for each field in the physical texture vector.

The fields in the physical texture vector 130 may be single-bit ormulti-bit fields. When multi-bit fields are used, the available quantumlevels can represent the probability that the physical object is thedesignated texture, is in a similar class of textures, or some othermeasure that signifies the relative strength that the representativetexture contributes to the physical texture of the original object.

Preferably, the physical texture vector will contain more than justtexture representations. For example, adjectives like shiny, dull,metallic, transparent, translucent, and reflective can be represented byfields within the physical texture vector.

FIG. 4 shows a functional diagram for the entire image search andretrieval system. The Image Repository 140 represents the physicalstorage environment that will be searched for the presence of an image.This repository 140 can be a local hard drive, a shared hard drive on anetworked computer, one or more file servers attached to a WAN or LAN,or an image storehouse as vast as the Internet. The Regions Of Interest(ROI) database 220 is a relational or analytical database that containsthe searchable vectors that represent the images stored in therepository 140. In order for an image to be searchable, it must containan entry within the ROI database 220.

Entries in the ROI database 220 are created by the image locator 160 andthe region of interest (ROI) classifier 170. These two modules work intandem to locate images within the repository 140 via an interface 150and extract the relevant information that will be stored in the ROIdatabase 220. The image locator 160 contains automated routines thatcontinually search the repository 140 for the existence of new imagery.Software utilized for searching can be web crawlers, spiders, or anyother type of intelligent agent that can continuously search anunstructured network 140 for the existence of new or updated files.

The ROI classifier 170 takes newly located images and splits them intoregions of interest using techniques including, but not limited tosegmentation, blob analysis, high frequency filter and Fouriertransforms. Most images will contain multiple regions of interest. Eachregion will be analyzed separately and will receive its own datastructure representation within the ROI database 220.

Graphical searches are performed by the collaborative workings of imageretrieval 190, the image search requestor 200 and the region of interest(ROI) query 210. Any graphical search request will be initiated eitherby an intelligent agent or by a user. The user search request canoriginate from inside or outside the network on which the search willtake place. The image search requestor is an abstraction layer thattranslates user or agent search requests into the language understood bythe ROI query 210.

As an example, assume that a graphical search user wishes to search foran “Irish Setter.” Since these are terms not utilized by the ROI query210, the image search requestor 200 may format the search request ascolor=reddish-brown, texture=furry, and shape=dog. Upon submitting thissearch request, the ROI query 210 will search the ROI database 220 for alist of the files most likely to contain the desired object. Upondetermining this list, the ROI query 210 will tell the image retrieval190 to retrieve the files from the repository 140. The resulting filesare retrieved via the interface 180 and returned to the image searchrequestor 200, which will format them for the user.

Another embodiment of the present invention eliminates the need for theimage retrieval 190 and the associated interface to the repository 180.In this embodiment, the image search requestor 200 would simply pass theimage file names to the user and allow the higher-level functionality toretrieve the desired files.

FIG. 4 shows a single ROI query 210, although this software is capableof handling multiple simultaneous requests from a plurality of imagesearch requestors 200. A preferred embodiment of the invention wouldhave multiple ROI query 210 modules, each running on the same machine,on different machines within the same facility, separate machines on thesame network, or separate machines on physically different networks.

Another preferred embodiment includes multiple ROI databases 220. It isnot necessary for all copies of the ROI database 220 to be in tightsynchronization. Since the ROI databases 220 are continuously beingmodified by the ROI classifier 170, the constraint of requiringsynchronized ROI databases 220 places unnecessary burdens on the system.

FIG. 5 shows the layout of a typical region of interest (ROI) datastructure 230 for an entry within the ROI database 220. Graphically theROI data structure 230 representation gives the impression that all ofthe multi-byte fields are the same length, but the fields are allvariable length. Every image that is analyzed by the ROI classifier 170will have at least one ROI data structure 230 within the ROI database220. In practice, there will be multiple regions of interest containedwithin each image, with each region of interest having a dedicated ROIdata structure 230 within the ROI database 220.

The file name field 235 contains the name of the file that contains theregion of interest, and the file location field 240 stores the fileslocation within the network. With a fluid medium like the Internet,graphical files can be changed frequently without any changes to thefile name or location. When an ROI data structure is initially created,the cache key is created for the image that is a semi-unique qualifierfor the image. The cache key field 245 retains the cache key for lateruse by the image locator 160 and the ROI query 210. Every time a file isretrieved from the repository the cache key is recomputed and comparedto the cache key field 245 in the ROI data structure 230. A mismatch inthe two values signifies that the image has been changed since the ROIclassifier 170 operated on the original image.

Regions of interest can be described in a multitude of types andformats. Examples of region of interest descriptions include, but arenot limited to bounding boxes, polygons, and masks. The ROI type field250 contains the type of the region of interest, and the ROI descriptorfield 255 contains the description of the region of interest that is ofthe format indicated by the type 250. Since the number of bytes neededto describe a region of interest is quite diverse, the ROI descriptorfield 255 can be a small as a few bytes and can be as large as severalkilobytes.

The next three fields in the ROI data structure 230 are the previouslydiscussed fields of the physical color vector 50, physical shape vector90 and physical texture vector 130. Other physical vectors are allowedwithin the data structure 230. Some possible options are the physicalsize vector 260, physical location vector 265 and the physical contentvector 270.

FIG. 6 shows the structure of a typical physical color vector 50, whichcontains a type field 280 that identifies it as a physical color vector50 and a length field 290 that identifies the length (in bytes) of thevector 50. The individual color spaces 40 from FIG. 1 are uniquelyrepresented by the color space fields 295 within the physical colorvector 50. It is possible for ROI data structures 230 to containphysical color vectors 50 that represent various versions of theindividual color spaces 40. The version field 285 identifies the versionof the physical color space 30 that was used to quantize the region ofinterest within the physical color vector 50.

FIG. 7 shows the structure of a typical physical shape vector 90, whichcontains a type field 300 that identifies it as a physical shape vector50 and a length field 310 that identifies the length (in bytes) of thevector 90. The individual shape descriptors are uniquely represented bythe shape descriptor fields 315 within the physical shape vector 90. Itis possible for ROI data structures 230 to contain physical shapevectors 90 that represent various versions of the individual shapedescriptors. The version field 305 identifies the version of thephysical shape descriptors that were used to quantize the region ofinterest within the physical shape vector 90.

FIG. 8 shows the structure of a typical physical texture vector 130,which contains a type field 320 that identifies it as a physical texturevector 130 and a length field 330 that identifies the length (in bytes)of the vector 130. The individual texture descriptors are uniquelyrepresented by the texture descriptor fields 335 within the physicaltexture vector 130. It is possible for ROI data structures 230 tocontain physical texture vectors 130 that represent various versions ofthe individual texture descriptors. The version field 325 identifies theversion of the physical texture descriptors that were used to quantizethe region of interest within the physical texture vector 130.

In a preferred embodiment of the present invention, the ROI query 210would be able to alter its search capabilities based on the versionscontained with the vectors 50, 90, 130 in the ROI data structure. Forexample, more advanced versions of a shape vector may includehigher-level descriptions of shapes, thus allowing more thoroughsearches based on higher-level shape constructs.

A preferred embodiment of the present invention allows the ROIclassifier 170 to utilize vector versions 285, 305, 325 to compare withthe latest versions of each of the vectors 50, 90, 130. When an image isretrieved by the image locator 160 that is already in the ROI database220, the ROI classifier 170 would check the version numbers of all ofthe vectors and, when a type of vector is not constructed with thelatest version, the ROI classifier 170 would reprocess all of theregions of interest for that image and update the vectors 50, 90, 130 tothe latest versions.

FIG. 9 shows a flow chart for the image locator 160. Each time the imagelocator function 340 finds a file, it checks 350 the ROI database 220for the existence of an ROI entry for that image. If the image is notrepresented, the image location is sent 355 to the ROI classifier 170for analysis. If the file is represented in the ROI database apseudo-unique cache key is created and compared to the entry 245 in theROI data structure 230. If the cache key compare fails, all ROI datastructures 230 associated with this image are invalidated 375 and thenew image is sent 376 to the ROI classifier 170. If the cache keycompare passes, the versions 285, 305, 325 of the vectors 50, 90, 130for all ROI data structures associated with this image are compared 380to the latest versions. If the versions are the most recent, thesoftware returns to locate another image 340. If one of the vectors isnot the most recent, the image is sent to the ROI classifier 170 to bereprocessed.

FIG. 10 shows a flow chart for the ROI classifier 170. When an image isreceived 390 from the image locator 160, it is processed to find aunique region of interest (ROI) 400. Upon locating a unique ROI, thecolor transform 410, shape transform 420, texture transform 430, andother defined transforms 440 are performed so the ROI can be describedwithin the physical spaces. The transforms will create 415, 425, 435,445 physical vectors 50, 90, 130, 275 for the ROI. Next the ROI datastructure 230 is constructed 450 and stored 460 in the ROI database 220.The original image is then tested 470 for the existence of anotherunique region of interest.

FIG. 11 shows an example of how a fuzzy logic color filter techniquecould be applied to a region of interest of an image in order to obtainthe physical color vector 50 associated with the actual color of thetrue object. In this example, the true object 510 parsed from an imageis an Exit sign with white and green RGB color values as shown. Variousexamples of image objects 522, 524, 526, 528, 530 and 532 are also showndepicting the associated RGB color data that might be presented by thecontent of an image. In this embodiment, a fuzzy logic color filtertechnique, such as described in U.S. Pat. No. 6,266,442, the disclosureof which is hereby incorporated by reference, could be used, althoughother techniques, alone or in combination may be useful with the presentinvention.

As shown in FIG. 12, an image histogram of image object 524 suggests apurple tint to the object 524. Preferably, the process of obtaining thephysical color vector 50 looks for known color combinations. The imageobject 524 represents the segmented object, the Exit traffic sign, withthe suspected purple cast removed. The ROI analysis incorporating thefuzzy logic color filter is used to determine that there is a fairlyhigh probability that if the foreground 540 is actually white, thebackground 542 is really green. Image processing for color tint removalis not actually performed. Instead, the software determines: IF color Ais white, what is the likelihood that color B is green? IF color A iswhite, what is the likelihood that color B is brown? IF color A iswhite, what is the likelihood that color B is red? IF color A is white,what is the likelihood that color B is orange?

In this embodiment, a Fuzzy Logic Color Filter (FLCF) is applied toobjects for all known or suspected color sets (combinations). The FLCFcan be applied to different color sets in parallel. The FLCF can beaccomplished with multiple processes on one CPU, or can be implementedwithin multiple CPUs within one box, or can be effected by a distributedprocessing approach.

The complete disclosures of the patents, patent applications andpublications cited herein are incorporated by reference in theirentirety as if each were individually incorporated. Variousmodifications and alterations to this invention will become apparent tothose skilled in the art without departing from the scope and spirit ofthis invention. It should be understood that this invention is notintended to be unduly limited by the illustrative embodiments andexamples set forth herein and that such examples and embodiments arepresented by way of example only with the scope of the inventionintended to be limited only by the claims set forth herein.

The invention claimed is:
 1. A computerized system for automatedretrieval of images from an unstructured repository, the computerizedsystem comprising: an image locator that automatically locates imagesfor analysis within the repository to populate a region of interest(ROI) database that contains searchable entries representing imagesstored in the repository; an ROI classifier that segments one or moreimages located by the image locator into a plurality of ROIs, each ROIcontaining image content associated with one or more actual features ofa physical object in the ROI, the ROI classifier having at least onecomputer-implemented transform filter adapted to extract one or morephysical descriptors from the image content for each ROI to be stored inthe ROI database as a physical feature vector, each physical descriptorcorresponding to at least one of the one or more actual features of agiven physical object in the ROI associated with the image content; andan ROI query module adapted to search the ROI database for imagescontaining content associated with at least one true physical object. 2.The system of claim 1 wherein the ROI database is selected from thegroup consisting of a relational database and an analytical database. 3.The system of claim 1 wherein the physical feature vector has asearchable data structure of physical features belonging to a physicalobject feature space representing a range of features of true objects.4. The system of claim 3, wherein the searchable data structure containsan ROI type and an ROI descriptor field contains a description of theROI having a format indicated by the ROI type.
 5. The system of claim 3,wherein the searchable data structure comprises at least one of aphysical color vector field, a physical shape vector field, or aphysical texture vector field.
 6. The system of claim 1, wherein theimage locator further comprises one or more automated routines forcontinually searching the repository for new images.
 7. The system ofclaim 1, wherein the region of interest classifier segments the one ormore images into regions of interest using one or more of segmentation,blob analysis, high frequency filter and Fourier transforms.
 8. Thesystem of claim 1 wherein, the ROI is selected from a set consisting ofa bounding box, a polygon, and a mask.
 9. The system of claim 1, whereinthe physical feature vector has a searchable data structure thatcomprises a physical color vector.
 10. The system of claim 9, whereinthe physical color vector belongs to a physical object color space thatdescribes a range of colors for the true objects that correspond to theimages stored in the repository.
 11. The system of claim 10 wherein thephysical color vector has a single-bit field.
 12. The system of claim 1,wherein the physical feature vector has a searchable data structure thatcomprises a physical shape vector.
 13. The system of claim 12, whereinthe physical shape vector belongs to a physical object shape space thatcontains two dimensional representations of true objects that correspondto the images stored in the repository.
 14. The system of claim 1,wherein the physical feature vector has a searchable data structure thatcomprises a physical texture vector.
 15. The system of claim 14, whereinthe physical texture vector belongs to a physical object texture spacethat describes a range of textures of the true objects that correspondto the images stored in the repository.
 16. A computer-implementedmethod for automated retrieval of images from an unstructuredrepository, the method comprising: automatically locating images foranalysis within the repository; populating a region of interest (ROI)database with the located images, the ROI database containing searchableentries representing images stored in the repository; segmenting thelocated images into a plurality of regions of interest (ROIs), each ROIcontaining content associated with one or more actual features of aphysical object; extracting at least one physical descriptor from thecontent for each ROI, each physical descriptor corresponding to one ormore of the actual features of a given physical object in the ROI; andstoring the at least one physical descriptor in the ROI database. 17.The method of claim 16, wherein segmenting the located images comprises:segmenting images located by the image locator into a plurality of ROIsusing one or more computer-implemented transform filters adapted toextract physical descriptors from the content for each ROI to be storedin the ROI database.
 18. The method of claim 17, wherein each physicaldescriptor is a data structure having a vector of physical featuresbelonging to a physical object feature space representing a range offeatures of true objects.
 19. The method of claim 18, wherein the vectorof physical features is one or more of a physical color vector, aphysical shape vector, and a physical texture vector.
 20. A tangible,non-transitory computer readable recording medium recorded with aprogram that, when executed by a processor, causes the processor toimplement a method for automated retrieval of images from anunstructured repository, the method comprising: automatically locatingimages for analysis within the repository; populating a region ofinterest (ROI) database with the located images, the ROI databasecontaining searchable entries representing images stored in therepository; segmenting the located images into a plurality of regions ofinterest (ROIs), each ROI containing content associated with one or moreactual features of a physical object; extracting at least one physicaldescriptor from the content for each ROI, each physical descriptorcorresponding to one or more of the actual features of a given physicalobject in the ROI; and storing the at least one physical descriptor inthe ROI database.
 21. The recording medium of claim 20, whereinsegmenting the located images comprises segmenting images located by theimage locator into a plurality of ROIs using one or morecomputer-implemented transform filters adapted to extract physicaldescriptors from the content for each ROI to be stored in the ROIdatabase.
 22. The method of claim 21, wherein each physical descriptoris a data structure having a vector of physical features belonging to aphysical object feature space representing a range of features of trueobjects.
 23. The method of claim 22, wherein the vector of physicalfeatures is one or more of a physical color vector, a physical shapevector, and a physical texture vector.